Skip to content

feat: add live integration testing setup and configurations - #545

Merged
techwritermat merged 14 commits into
masterfrom
docs-snippets-live-integration-tier
Nov 19, 2026
Merged

techwritermat merged 14 commits into
masterfrom
docs-snippets-live-integration-tier

Conversation

@techwritermat

Copy link
Copy Markdown
Contributor
  • Updated .gitignore to include environment files for live integration tests.
  • Enhanced .mocharc.json to exclude live integration tests from general test runs.
  • Added multiple scripts in package.json for running various live integration tests.
  • Introduced GitHub Actions workflow for live integration testing, ensuring secure handling of secrets and scheduled runs.
  • Implemented network blocking in test setup to prevent real network calls during tests.
  • Added new test cases for heartbeat failure reporting and HTTP status code handling.
  • Created example environment files for live integration tier configuration.

- Updated .gitignore to include environment files for live integration tests.
- Enhanced .mocharc.json to exclude live integration tests from general test runs.
- Added multiple scripts in package.json for running various live integration tests.
- Introduced GitHub Actions workflow for live integration testing, ensuring secure handling of secrets and scheduled runs.
- Implemented network blocking in test setup to prevent real network calls during tests.
- Added new test cases for heartbeat failure reporting and HTTP status code handling.
- Created example environment files for live integration tier configuration.
@pubnub-snyk-universal-prd

pubnub-snyk-universal-prd Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

✅ Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
✅ Open Source Security 0 0 0 0 0 issues
✅ Licenses 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

- Disabled the docs-snippets testing suite, indicating that samples now run in the live tier.
- Updated the live integration workflow to allow testing on pull requests from branches within the same repository, while maintaining restrictions on forked pull requests.
… state verification

- Updated multiple test files to use `settled` instead of `eventually` for assertions that require consecutive reads to confirm state consistency.
- Enhanced comments to clarify the need for consecutive reads in presence, data synchronization, and message action tests.
- Adjusted README and coverage files to reflect the changes in testing methodology.
…anation

- Introduced `createWithRunUniqueId` for creating DataSync objects with unique IDs, ensuring retries handle transient errors correctly.
- Updated multiple test files to utilize `seedOnFreshChannel` for seeding channels and messages, improving test stability.
- Added `explainSdkError` functionality to provide detailed error messages for PubNub SDK failures in test outputs.
- Enhanced README documentation to reflect new testing methodologies and error handling improvements.
… reliability

- Introduced `removeAfterTest` functionality to register the removal of DataSync objects created during tests, ensuring they are cleaned up regardless of test outcomes.
- Updated multiple test files to utilize `removeAfterTest`, improving test isolation and preventing leftover objects from affecting subsequent tests.
- Added `cleanupAfterTest` and `runCleanups` methods to manage cleanup operations efficiently after each test.
- Enhanced README documentation to reflect the new cleanup methodology and its benefits for test reliability.
…gnostics

- Added waiting mechanisms to ensure that DataSync reads return expected results after create and delete operations, improving test reliability.
- Introduced detailed diagnostics for capturing statuses and events during DataSync operations, aiding in troubleshooting.
- Updated multiple test files to utilize new waiting and diagnostic features, ensuring consistent state verification across tests.
- Enhanced comments for clarity on the purpose of waiting for state stabilization in tests.
- Deleted the docs-snippets directory and its associated files to streamline the project structure.
- Updated .codacy.yml to exclude paths related to the removed docs-snippets.
- Adjusted .gitignore and .npmignore to reflect the removal of docs-snippets.
- Enhanced package.json scripts for integration tests by refining linting commands and removing references to docs-snippets.
- Updated live integration workflow to ensure compatibility with the new structure.
- Eliminated the test for documentation snippets from the CI workflow to streamline the testing process.
- Updated the coverage.json file to clarify the description of the legacy decrypt functionality.
Comment thread .github/workflows/live-integration.yml Outdated
live:
name: Live tier
needs: [offline]
# TEMPORARY: also runs on pull requests from a branch of this repository, so the tier can be tested before it is merged

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it mean that this will not be scheduled for draft PR?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the workflow will run on non-draft PRs to master

- Added support for grouping live integration tests into distinct API groups, enhancing organization and execution control.
- Implemented `groups.cjs` to define groups, roles, and associated test files, ensuring each test file belongs to exactly one group.
- Updated Mocha configurations to utilize group specifications, allowing targeted test execution via the `LIVE_GROUP` environment variable.
- Enhanced the CI workflow to accommodate group-based testing, improving test management and reporting.
- Refined README documentation to explain the new grouping structure and its implications for running tests.

@parfeon parfeon left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing critical but still will need some attention.

Comment thread .github/workflows/live-integration.yml Outdated
- 'test/integration/endpoints/time.test.ts'
- 'test/integration/components/subscription_manager.test.ts'
- 'package.json'
- '.github/workflows/live-integration.yml'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This basically restrict runs only for the cases, when files at specified paths has been changed and won't run for any behavioral / interface changes.
Is it intentional that we don't want to run tests when client code changes?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not intentional anymore. I removed the paths filter in 68cb650. The live tests now run for every non-draft pull request into master, including changes under src/. Each push to the pull request starts a new run. A full run takes about 10 minutes, one group at a time on the shared keysets.

Comment thread .github/workflows/live-integration.yml Outdated
# never interleave. A running run is never cancelled. Runs that get no live jobs (draft or fork pull requests) take a group of
# their own, so they never wait and never replace a pending live run.
concurrency:
group: ${{ (github.event_name == 'pull_request' && (github.event.pull_request.draft || github.event.pull_request.head.repo.full_name != github.repository)) && format('live-tests-no-live-{0}', github.run_id) || 'docs-snippets-live' }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

format('live-tests-no-live-{0}', github.run_id) - open gates to malicious intentions where the fork owner may intentionally make 100 different pushes to the PR branch, and each will be started because each will have a unique github.run_id and group will be unique for each run. This could be crucial if there will be a self-hosted runner somewhere in the chain in the future.

Why not keep group as simple as ${{ github.workflow }}-${{ github.ref }}?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, you are right. Runs that get no live jobs (draft and fork pull requests) now share one group per pull request, keyed on github.ref, and cancel-in-progress is on for them (68cb650). Many pushes to one fork branch now leave one run, not one per push. I did not use ${{ github.workflow }}-${{ github.ref }} for the live runs. Two pull requests would then run live jobs on the shared keysets at the same time.

Comment thread .github/workflows/live-integration.yml Outdated
# their own, so they never wait and never replace a pending live run.
concurrency:
group: ${{ (github.event_name == 'pull_request' && (github.event.pull_request.draft || github.event.pull_request.head.repo.full_name != github.repository)) && format('live-tests-no-live-{0}', github.run_id) || 'docs-snippets-live' }}
cancel-in-progress: false

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we sure that we don't want to cancel previous test run? What if you someone made a quick fix, but now will have to wait because there is active workflow run or go and cancel it manually?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I kept cancel-in-progress: false on the live group. The group is global, so true would also cancel the live run of an unrelated pull request. To cover your quick-fix case, I added cancel-superseded-live-runs.yml (68cb650). On each push, it cancels the older live runs of the same pull request and nothing else. It is a separate workflow because a run that waits for the group cannot run a step. A cancelled run still runs its cleanup step, and the next run starts with its own cleanup pass, so a cancel does not break the next run. The new workflow has not run on GitHub yet. I will check it on the next push.

Comment thread .github/workflows/live-integration.yml Outdated
with:
persist-credentials: false
- name: Setup NodeJS 22
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious on whether we actually want to pin by commit action maintained by GitHub themselves? Isn't it a sign of "trusted" action?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair question. I switched actions/setup-node to the official tag, @v4 (68cb650), like actions/checkout@v7. For actions that GitHub publishes, a tag is enough. A commit pin matters more for third-party actions, and this workflow has none.

- Revised the README to clarify the keyset and API key requirements, enhancing the understanding of necessary configurations for live integration tests.
- Removed outdated comments in test files to streamline the code and improve readability.
- Added a new section in the README detailing keyset roles and their associated requirements, ensuring comprehensive documentation for users.
- Refined the type definition for the global setTimeout function in channel tests to enhance type safety.
- Updated ESLint comments across multiple files to include both no-require-imports and no-var-requires rules, improving code quality and adherence to TypeScript best practices.
- Introduced a new GitHub Actions workflow (`cancel-superseded-live-runs.yml`) that cancels older live test runs for the same pull request when a new push occurs, ensuring that only the most recent changes are tested.
- Updated the `live-integration.yml` workflow to clarify its behavior regarding concurrency and cancellation of older runs.
- Revised the README to explain the cancellation process and its implications for live test runs, enhancing documentation clarity.
@techwritermat
techwritermat merged commit 3c98e8a into master Nov 19, 2026
19 checks passed
@techwritermat
techwritermat deleted the docs-snippets-live-integration-tier branch November 19, 2026 10:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants